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Overview 


SSIP Phase | plans require states to determine a “state- 
identified measurable result” (SIMR) through a systematic 
process that is summarized graphically through a 

Theory of Action (ToA). A ToA sets forth assumptions or 
hypotheses about how an improvement strategy works. 
These hypotheses are then tested during an evaluation. 
The power of aToA is in its specificity of thought and in 
the explicit reasoning that calls attention to the essential 
steps and checkpoints in implementing an improvement 
strategy. Thus, the ToA establishes a clear path toward 
the end goal of affecting child and family outcomes. 
Importantly, the ToA acts as a set of checkpoints to 
evaluate if the expected outcomes at each step along the 
way are realized and, if they are not, suggests where to 
intervene to get the improvement strategy back on track. 


This paper demonstrates how the work completed 
under SSIP Phase | contributes to a key component of 
the activities required during Phase Il: an evaluation 
plan. By using the ToA as an outline or a roadmap for 
the evaluation plan, state SSIP leaders can identify 

and measure, through performance indicators, the 
checkpoints necessary for marking progress toward the 
SIMR. The first two sections of this white paper illustrate 
the relationship between SSIP Phase | and Phase II and 
briefly demonstrate how the ToA can be used to develop 
the evaluation plan and formulate evaluation questions. 
The third section shows in detail how SSIP leaders can use 
aToA to develop performance indicators that measure 
progress toward the SIMR. 


The Relationship Between SSIP Phase | and Phase II 


Planning an evaluation and conducting an evaluation 

are companion processes. Unfortunately, different 
audiences use different terms for similar concepts. The 
resulting confusion undermines integration of planning 
and evaluation, so it is important to make the connection 
between the different terms. As illustrated in Exhibit 1, the 
SSIP evaluation plan (Phase Il), directly proceeds from the 
activities conducted in Phase |; that is, from 


e Ananalysis of student outcomes, including drilling 
deep into the data to understand the performance of 
disaggregated student groups (e.g., ages, disability, type 
of outcome, location); 
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¢ An analysis of infrastructure strengths and weaknesses, 
including governance, fiscal, technical assistance, and 
monitoring systems; 


Discussions by stakeholder groups, who thoughtfully 
frame an end result for the program's efforts, called a 
SIMR; and 

Identification of coherent improvement strategies 
that will build state and local education agency (LEA) 
capacity to achieve the SIMR. 
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Exhibit 1. Linking SSIP Phase | to SSIP Phase II 


Improvement 
Focus Area to 
Address SiIMR 


THEORY 


OF ACTION 


Startegies 
& Activities 
(Inputs/Outputs) 


Research 
Questions 


EVALUATION 


PLAN 


The state’s response to improving the SIMR is articulated 
graphically in a ToA and can be specified as strategies, 
actions, tactics, or a host of other terms. The strategies 
generate potential evaluation questions, provide insights 
on the program's activities and some or all of the targeted 
audiences, make explicit short-term and intermediate 
outcomes, and clarify the long-term results desired by 
the program. 


Exhibit 2. Hierarchy of Possible Outcomes 
——»SYSTEMS AND ENVIRONMENT CHANGE 


CHILD AND FAMILY OUTCOMES 
Learning, social-emotional, behavioral child- and 
family-level end results 


ACTIONS 
New skills used and patterns of behavior adopted 
by target audiences 


LEARNING 
New knowledge and skills as end results 


REACTIONS 
Readiness; degree of interest toward the program; 
acceptance of activities, and of education methods 


PARTICIPATION 
Number of people reached; characteristics of the 
frequency, intensity, and quality of contact 


Source: Adapted from CDC (1999) 


— 
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| 
( 
| 


Infrastructure Data 
Analysis Analysis 
Short-term Long-term 

Intermediate Outcome 
Outcomes (SIMR) 


Evaluation plans are strengthened by collecting evidence 
of impact at several levels or steps in an outcomes 
hierarchy; information on outcomes from the proximal 
steps helps to explain outcomes at the distal steps, which 
take longer to achieve. All steps in the hierarchy are 
necessary outcomes and demonstrate progress toward 
the results-oriented outcome, the SIMR (see Exhibit 2). 


To illustrate, a program’s initial intended aim is to obtain 
participation among targeted staff and positively 
influence their reactions to the program activities that 
affect their learning (e.g., knowledge, opinions, skills). 
Through this learning, people and organizations take 
actions that result in behavioral and social change, 
including implementing the program with fidelity in the 
relevant setting and children and families demonstrating 
continued progress toward the expected outcome within 
the established timeframe. Concurrently, systems and 
environmental changes in conditions are the result of 
recommendations, actions, policies, and practices.' 


1 Evaluating the progress of change in the infrastructure is beyond the scope 
of this paper. ECTA/DaSy Centers have an excellent tool to use indicators as 
a means of tracking progress of the seven infrastructure components (see 
The DaSy Center. (2014). DaSy Data Systems Framework. Menlo Park, CA: 
SRI International). 
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Using a ToA to Develop an SSIP Evaluation Plan 


When developing the SSIP evaluation plan, SSIP state 
leaders need not start from scratch. Instead the ToA 
provides an initial direction for the evaluation plan. 


Exhibit 3 illustrates how a ToA for a multi-tiered system of 


supports (MTSS) strategy conveys three critical features of an 


evaluation plan: outcomes, target audiences, and activities. 


Exhibit 3. Theory of Action as an Outline for an Evaluation Plan 


schools in target districts 
to assist in establishing 
robust multi-tiered systems 
of support... 


Then the LEA 
Key Strands (e.g., teachers and Then teachers/ 
of Action If the SEA administrators) support teams So that 
Multi-tiered | ...provides professional ...Will increase its capacity | ... monitor students with Reading 
systems of development opportunities | to implement MTSS disabilities more closely in achievement for 
support and coaches to elementary | practices with fidelity... terms of their academic and students with 


behavioral progress and needs | disabilities in 
elementary grades 
in target districts 
will improve. 


...and more quickly and 
effectively provide supports 
and interventions that help 
students with disabilities be 
successful academically and 
behaviorally.... 


Note: SIMR Outcome in BOLD; targeted audiences in Italics; and activities are underlined; SEA = state education agency. 


Moving from right to left in the ToA above, the SIMR 
outcome defines what needs to be achieved and is an 
ambitious and Jonger term outcome: 


Increase the percentage of students with 
disabilities in schools in target districts that score 
at least Basic on the state achievement test for 
grades 3-5 in reading. The number of students with 
disabilities scoring at least Basic will increase by 3 
percentage points per year from the baseline score 
percentage. This rate of improvement constitutes 
an ambitious yet achievable goal that will 
ultimately raise the percentage of students with 
disabilities scoring Basic or higher by 15 percentage 
points over 5 years. 


There are targeted audiences that the program needs to 
urge into action in order to make progress on addressing 
these educational challenges. Generally, district and 
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school, or early childhood and program staff need to 
take action. Some action may be needed by the families 
and communities. Some actions may be necessary by 
constituencies all across the state infrastructure. And 
finally, teachers and service providers must instruct, 
engage, and progress- monitor children in the program 
to produce the favorable child-level outcomes (the SIMR). 
In the ToA above, target audiences are at the state, LEA, 
school, and classroom levels. 


It is the agency's role to specify the strategies or 
activities required to be in place to achieve the outcomes 
in the targeted groups. Activities will vary with the 
audience and program. In the ToA above, the coherent 
improvement strategy is a robust system of multi-tiered 
supports. The specific activities related to the strategy 
may include providing professional development to 
elementary schools in target districts (SEA- and LEA-level 
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activity), providing coaches in schools in target districts 
(SEA- and LEA-level activity), establishing robust systems 
of multi-tiered supports in schools (LEA- and school-level 


activity), monitoring students and providing supports 
(classroom-level activity). 


Using a ToA to Develop Evaluation Questions 


To be effective, an evaluation plan needs focus. This is 
where evaluation questions come in. There is no specific 
formula for writing evaluation questions; however, they 
must be asked in a manner that requires a response that 
can be accurately measured, analyzed, and reported. 
Given that in many instances resources available for 
evaluations are limited, what do SSIP leaders and their 
stakeholders really need to know to understand how 
well the program's activities are being implemented and 
if the program has the expected impact on the target 
audiences and ultimately the SIMR population? Because 
the program activities and desired outcomes are made 
transparent in the TOA, it is easy to use it to develop 
evaluation questions. 


Continuing with the example above, by aligning the 
evaluation questions to the hierarchy of outcomes, 
possible evaluation questions of the installation of MTSS 
practices include: 


1. Participation outcome. What proportion of teachers 
who were identified for receipt of the MTSS training 
participated? What proportion of the intended hours 
of professional development did teachers receive? 
What proportion of the intended hours of coaching did 
teachers receive? 


2. Reaction outcome. To what extent do the teachers 
agree that the MTSS practices can easily be 
implemented into their classroom instruction? To what 
extent do teachers report their readiness for using the 


2 From the IDC Interactive Institutes 2015, resources are available on 
writing evaluation questions. http://dasycenter.sri.com/downloads/ 
DaSy_conferences/2014/DaSy_W04_QuestionsDataReports_ 
ConnectingDataDots_FINAL090814_Acc2.pptx 


www.ideadata.org 


MTSS practices? in their classroom and school? To what 
extent do teachers report the practices will improve 
student outcomes? 


Learning outcome. How well do teachers report that 
they understand how the MTSS practices influence 
student outcomes? To what extent do teachers rate 
their effective use of the MTSS practice highly? 


. Action outcome. To what extent do teachers 


implement the MTSS practices with fidelity? To what 
extent do the teachers instruct students with the 
intended dosage and frequency? 


. Child and family outcome(s): 


a. What is the impact of MTSS on the reading 
achievement of students with disabilities in 
schools in target districts compared to the reading 
achievement of students with disabilities in 
comparison schools? 


b. What is the impact of MTSS on the classroom 
behavior of students with disabilities in schools 
in target districts compared to the behavior of 
students with disabilities in comparison schools? 


c. What is the impact of MTSS on the reading 
achievement of all students in target schools 
compared to the reading achievement of all 
students in comparison schools? 


3 In these evaluation questions, MTSS practices represent the program to 
influence student outcomes. The data collection instruments, however, 
would include questions that address each MTSS practice. 
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Identifying Performance Indicators 
of Progress for the Evaluation Plan 


Identifying performance indicators helps guide and 
monitor progress toward the long-term outcome, or in the 


case of SSIP, the SIMR, and is important for several reasons: 


(1) When a new or large initiative is getting started, there 
may be clarity about the “big change” the program is to 
produce, but little else. (2) It is easy to place too much 
emphasis on listing program activities, many of which 
either cannot be measured or the measurement data 
necessary would overwhelm the evaluation and become 
burdensome to collect and costly to use. (3) Because 

the components of educational programs are often 
expressed in global or abstract terms, indicators—specific, 
observable, and measurable statements—help define 
exactly what we mean or are looking for and require 

the use of such terms as number of, percent of, mean of, 
or similar phrases. By working backward from the end 
outcomes and asking “how to,’ SSIP leaders can identify 
what will be involved in producing change. 


Indicators provide clear definitions of global outcome 
statements such as “Teachers measure student reading 
progress twice a week” or “Families adopt in-home literacy 
techniques.’ The teacher-based indicator might specify 
the type of reading intervention, the duration, or the 
adherence to the intervention’s fidelity, for example: 90 
percent of teachers will measure oral reading fluency 

of students in Tier 3 once a week. Likewise, the family 
indicator might specify the in-home techniques and the 
intensity or duration of their adoption, such as, all families 
will read to their children at least five times a week for 

20 minutes each time. Indicators such as these provide 
clearer definitions of the global statement and help SSIP 
leaders understand how to measure progress toward the 
end outcome. 


The use of indicators originates from a type of evaluation 
called performance measurement and can have many 
meanings. For our purpose, it is defined as regular 
measurement of the implemented service delivery and 
results (outcomes). Regular measurement of progress 


www.ideadata.org 


toward specified outcomes is a vital component 

of any effort at managing for results. Performance 
measurement is a stakeholder-oriented process 

that focuses on maximizing benefit and minimizing 
negative consequences for the participants (Hatry, 

2006). Specifically, a performance indicator is: an 
observable measure of both the process and the outcome, 
at the child, classroom, teacher, family, school or district 
level, which identifies a specific numerical measurement 
indicating progress toward the outcome. As noted earlier, a 
performance indicator usually begins with words, such as 
number of, percent of, ratio of, proportion of, or mean, or 
another similar phrase. 


There are three levels or types of performance indicators, 
and each one contributes to marking progress toward the 
SIMR. Each one is defined below. 


Types of Performance Indicators 


e Indicators of short-term outcomes: These outcomes 
are expected to lead to the desired end but are not 
themselves ends. There may be multiple short-term 
outcomes. Linking to the hierarchy of outcomes in 
Exhibit 2, examples of indictors of short-term outcomes 
include Participation outcome—percentage of teachers 
who rate highly the usefulness and feasibility of the 
program; Reaction outcome—percentage of teachers 
who express a high interest toward the program; 
Learning outcome—percentage of teachers who self- 
rate their positive changes in knowledge about the 
practice. 

¢ Indicators of intermediate outcomes: Again, 
linking the intermediate indicators to the hierarchy 
of outcomes in Exhibit 2, action outcome examples 
include, demonstrating the new practice or skill, 


4 Outputs from activities are not explicitly stated in these descriptions 
(e.g., number of courses offered, number or percentage of staff completing 
course). Managers and evaluators may elect to identify the outputs as 
part of a logic model sequence, but in this paper, the emphasis is on the 
indicators required for marking progress toward the SIMR (changes in 
behavior based on skills and knowledge). 
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including measures of the dosage, frequency, or quality on child and family outcomes such as learning, 

of services,° and are direct results of the improvement social-emotional, behavioral child- and family-level 
activities. Examples include frequency of MTSS end results. The indicator will specify to what extent 
meetings by teachers and administrators or percentage students or families will increase their academic or 

of educators observed implementing MTSS practices functional performance, by how much, and over what 
(e.g., continually progress-monitor students’ behavior time period. 

and academic progress, provide supports across 


Linking the outcomes and performance indicators is 
essential to marking or evaluating progress toward 
the intended results set forth by SSIP leaders. In the 
detailed example below, a state has identified that a 
subset of districts have large literacy achievement gaps 
See eee eran is (ier hen hee meee, between students with disabilities and thelr peers in the 
2005). early grades (see Exhibit 4). 


the tiers). 


e Indicators of long-term outcomes: These are the 
end results sought. For SSIP, this end result focuses 


Exhibit 4. A Hypothetical State Rationale of a Theory of Action 


SIMR: Improve reading achievement for elementary-age students with disabilities in a subset of districts with large 
achievement gaps between students with disabilities and their peers without disabilities. 

Data and root cause analysis: Our data analysis revealed that the achievement of students with disabilities in reading 
(as measured by the statewide assessment) in the elementary grades was significantly below that of their peers without 
disabilities. The achievement gap, in terms of the percentage of students proficient in reading, was highest in grades 3, 4, 
and 5. While there were variations and differences in achievement by students’ disability categories, race/ethnicity, and 
socioeconomic status, our stakeholder group (which included representation from local special education directors, special 
education teachers, general education teachers, school administrators, family advocacy groups, and higher education) 
was compelled by the overall level of low achievement in the elementary grades. They also noted that certain districts in 
the state had greater achievement gaps than others. Therefore, our state chose to focus its SIMR on improving reading 
achievement in the elementary grades in a subset of districts with the largest achievement gaps. 


The state's root cause analysis revealed a strong relationship between students’ educational environments and their 
academic achievement as measured by the statewide assessment. Specifically, elementary-age students educated in 
substantially separate placements as well as inside the regular class less than 40 percent of the day scored lower than 
similar students with disabilities educated in the regular classroom more than 80 percent of the day. In addition, a 
statewide survey of MTSS implementation revealed that districts reporting full implementation of an MTSS system also had 
higher rates of students with disabilities scoring proficient on the statewide assessment in reading than those reporting 
lower levels of implementation. Finally, our stakeholder group felt that poor instructional quality and lack of opportunities 
for general and special educators to co-teach were factors in the low achievement of some students with disabilities. 
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Based on the results of its root cause analysis, the state 
has identified a multi-pronged approach to improve 
academic outcomes for students with disabilities in 
reading in elementary schools in target school districts. 
This multi-pronged approach includes establishing 
robust MTSS in the elementary schools in target districts, 
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increasing access to the general education curriculum for 
students with disabilities, improving instructional quality, 
and encouraging co-teaching models (Exhibit 5). For the 
purposes of illustration, short-term, intermediate, and 
long-term outcomes and indicators are specified for just 
one of these strategies—MTSS (see Exhibit 6). 
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Exhibit 5. Example of a Multi-Pronged Theory of Action 


Illustrative Example of a Theory of Action 


and coaches to elementary 
schools in target districts 
on research-based 
instructional strategies in 
literacy... 


to meet the individual 
needs of students with 
disabilities... 


Key Strands Then the LEA (teachers Then teachers 
of Action If the SEA and administrators) and support staff 
Multi-tiered ...provides professional ...Will increase its capacity | ...will monitor students with 
systems of development opportunities | to implement MTSS disabilities more closely in 
support and coaches to elementary | practices with fidelity... terms of their academic and 
schools in target districts to behavioral progress and needs 
assist in establishing robust 
multi-tiered systems of ...and will more quickly and 
support... effectively provide supports 
and interventions that help 
students with disabilities be 
successful academically and 
behaviorally... 
Educational ...provides professional ...will make more informed | ...will provide students with 
environments | development opportunities | and intentional decisions disabilities access to the 
and coaches to elementary | about educational general curriculum.... 
schools in target districts to | placements for students 
explore data on educational | with disabilities... 
environment and analyze 
where students with 
disabilities are receiving 
their instruction... 
Instructional | ...provides professional ..Will implement evidence- | ...will deliver high-quality 
quality development opportunities | based practices with fidelity | instruction .... 


Co-teaching 


...provides professional 
development opportunities 
and coaches to elementary 
schools in target districts 
to establish co-teaching 
models... 


...will offer general and 
special educators the 
opportunity to work more 
closely together in the 
classroom to design lessons, 
support students, and braid 
best instructional strategies 
for students with disabilities 
and their peers... 


...and will benefit from 

the shared knowledge and 
expertise of their colleagues 
in general and special 
education... 
provide students with 
disabilities access to the 
general curriculum.... 


Will 


So that 


Reading 
achievement for 
students with 
disabilities in 
elementary grades 
in target districts 
will improve. 
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Exhibit 6. Illustrative Example of a (Partial) Theory of Action Linked With 
Performance Indicators 


multi-tiered systems of 
support... 


Key Strands 
of Action If the SEA 
Multi-tiered ...provides professional 
system of development opportunities 
supports and coaches to elementary 


schools in target districts to 
assist in establishing robust 


Then the LEA (teachers 
and administrators) 
...will increase its capacity 


to implement MTSS 
practices with fidelity... 


Then teachers 
and support staff 


...Will monitor students with 
disabilities more closely in 
terms of their academic and 
behavioral progress and needs 


...and will more quickly and 
effectively provide supports 
and interventions that help 
students with disabilities be 
successful academically and 
behaviorally... 


So that 


Reading 
achievement for 
students with 
disabilities in 
elementary grades 
in target districts 
will improve. 


! 


! 


Indicators of 
Short-term Outcomes 


Participation Outcomes: 

e Percentage of participants who 
rated professional development 
opportunities as high quality, 
relevant, and useful 

e Number of additional coaches 
hired 

e Frequency of visits by coaches 
to elementary schools in target 
districts 

e Percentage of participants who 
rated coaching sessions as high 
quality, relevant, and useful 


Reaction Outcome: 


e Percentage of teachers reporting 
an interest in learning the program 

e Learning Outcomes 

e Percentage of participants who 
reported increased knowledge 
about effective MTSS systems and 
practices 

e Percentage of educators who 
reported increases in their 
application of knowledge and skills 
obtained through professional 
development 


Indicators of Intermediate Outcomes 


Action Outcomes: 


e Percentage of school sites at the installation 
stage of MTSS system and practices 

e Percentage of school sites that developed MTSS 
systems frameworks 

e Frequency of MTSS meetings by teachers and 
administrators convened at least once a month 
to review data profiles of classes and students 

e Percentage of educators who self-reported 
implementing MTSS practices (e.g., continually 
progress-monitor students’ behavior and 
academic progress, provide supports across 
Tier I, Tier Il, and Tier III) 

e Percentage of educators observed implementing 
MTSS practices (e.g., continually progress-monitor 
students’ behavior and academic progress, 
provide supports across the Tiers) 

e Percentage of school sites demonstrating 
evidence of robust MTSS systems in elementary 
schools in target districts, including existence 
of high-quality Tier Il and Tier Ill 

e Decrease in the percentage of students 
receiving Tier Ill and Tier Il supports 

e Changes in the numbers and percentage of 
students receiving supports at each level over a 
set period of time as a proxy for demonstrating 
use of MTSS data-based management practices 


Indicators of 
Long-term Outcomes 


Decrease in number of disciplinary 
referrals 

Improvements in the academic 
growth in reading of students with 
disabilities 

Improvements in the academic 
growth of any students receiving 
Tier Il and Tier Ill supports 
Decrease in number of students 
referred or identified for special 
education 
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Summary 


This paper demonstrates how the activities completed 
under SSIP Phase | contribute to the evaluation planning 
component of SSIP Phase Il, which then guides the 
evaluation work conducted in subsequent years. The 
ToA, as the product of data and infrastructure analysis, 
stakeholder discussions, and SIMR identification, visually 
depicts the underlying reasoning behind the choice of 
improvement strategies adopted by a state to achieve 
its long-term outcome (SIMR). Moreover, the ToA clearly 
defines the valued outcome(s), target audiences, and 
activities that a state will engage in over the life of its 
SSIP. Through this graphical argument, SSIP leaders and 
their stakeholders can generate important evaluation 
questions at each level of the outcomes hierarchy, 
identify essential steps and checkpoints in implementing 
selected improvement strategies, and evaluate if the 
expected outcomes at each step along the way are 
realized. 


SSIP evaluation plans are strengthened by periodic 
collection of evidence of impact at each level in an 
outcomes hierarchy. Regular measurement of progress 
toward specified outcomes is a vital component of any 
effort at managing for result as it assures SSIP leaders 
that the system is on track or indicates that mid-course 
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changes may be necessary. Performance indicators are 

a common method of collecting this evidence as they 
are written in specific, observable, and measurable 
language to define exactly what SSIP leaders mean or 
are looking for and require the use of such terms as 
number of, percent of, mean of, or similar phrases. This 
paper describes three types of performance indicators. 
Indicators of short-term outcomes measure participation, 
reaction, and learning outcomes in the hierarchy of 
outcomes. Indicators of intermediate outcomes measure 
action outcomes in the hierarchy of outcomes. Finally, 
indicators of long-term outcomes measure the end 
results sought, which for the SSIP focuses on child and 
family outcomes such as learning, social-emotional, 
behavioral child- and family-level end results. 


Woven throughout this paper is a practical illustration 
of how to use a ToA to develop evaluation questions 
and performance indicators linked to an outcomes 
hierarchy. While the example in the paper is focused on 
a single component of a multi-pronged ToA, the process 
outlined can be replicated to guide SSIP leaders through 
some of the steps necessary to develop the evaluation 
component of SSIP Phase II. 
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